Rectified Factor Networks
نویسندگان
چکیده
We propose rectified factor networks (RFNs) as generative unsupervised models, which learn robust, very sparse, and non-linear codes with many code units. RFN learning is a variational expectation maximization (EM) algorithm with unknown prior which includes (i) rectified posterior means, (ii) normalized signals of hidden units, and (iii) dropout. Like factor analysis, RFNs explain the data variance by their parameters. For pretraining of deep networks on MNIST, rectangle data, convex shapes, NORB, and CIFAR, RFNs were superior to restricted Boltzmann machines (RBMs) and denoising autoencoders. On CIFAR-10 and CIFAR-100, RFN pretraining always improved the results of deep networks for different architectures like AlexNet, deep supervised net (DSN), and a simple “Network In Network” architecture. With RFNs success is guaranteed.
منابع مشابه
Flexible Rectified Linear Units for Improving Convolutional Neural Networks
Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU). FReLU improves the flexibility of ReLU by a learnable rectified point. FReLU achieves a faster convergence and higher performance. Furthermore, FReLU does not rely on strict assumptions by s...
متن کاملDReLUs: Dual Rectified Linear Units
Rectified Linear Units (ReLUs) are widely used in feed-forward neural networks, and in convolutional neural networks in particular. However, they can be rarely found in recurrent neural networks due to the unboundedness and the positive image of the rectified linear activation function. In this paper, we introduce Dual Rectified Linear Units (DReLUs), a novel type of rectified unit that comes w...
متن کاملEmpirical Evaluation of Rectified Activations in Convolutional Network
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our expe...
متن کاملNonlinear activations for convolutional neural network acoustic models
Following their triumphs in visual recognition tasks, convolutional neural networks (CNNs) have recently been used to learn the emission probabilities of hidden Markov models in speech recognition. The key distinction of CNNs over deep neural networks (DNNs) is that they leverage translational invariance in the frequency domain, such that weights are shared and there are significantly fewer par...
متن کاملA limit on the effect of rectified diffusion in volcanic systems
[1] Forced oscillations can push dissolved volatiles into bubbles by a process called rectified diffusion. In engineering applications, the pumping action of rectified diffusion makes bubbles grow. In the geosciences, rectified diffusion is a suggested mechanism to trigger volcanic eruptions with seismic waves generated by distant earthquakes. Previous geoscience studies adopted the engineering...
متن کامل